1. Field of the Invention
The present invention relates to a video communication system wherein a picture is sent via a transmission line on which errors can be generated during sending through a wireless transmission line, a LAN, etc., a video communication method, and computer readable recording media on which a video communication program is recorded.
This application is based on Japanese Patent Application, No. Hei 10-49094 and Japanese Patent Application, No. Hei 10-49095, the contents of which are incorporated herein by reference.
2. Description of the related Art
In the communication of video data, the inter-frame prediction encoding method is a representative video encoding method for increasing compressibility. In this inter-frame prediction encoding method, at the sending side, the predictive error between the latest previously encoded picture and the present picture is encoded, and the only this part is sent as encoded data via the transmission line to the receiving side. Then, at the receiving side, using a picture previously received, decoded and then stored as a reference picture, the present picture is decoded from this reference picture and the received data.
When encoded data using this inter-frame prediction encoding is transmitted, if an error is carried into the encoded data over the transmission line, the picture will be damaged and moreover the error will propagate to the picture of the later encoded data. That is, when the receiving side uses erred picture as a reference picture, even if encoded data of a subsequent picture does not have an error, the error will propagate to the decoded picture because the area damaged by the error is referenced.
One technology for resolving this problem is a method wherein the sending side refers only to a picture which can be correctly decoded at the receiving side, as in the Annex N of the picture encoding method ITUxe2x80x94T H. 263. In this method, a reference picture memory for a plurality of pictures is provided at the sending side and receiving side, and furthermore, the sending side decodes the reference picture by using a picture designated at the receiving side. The encodingxe2x80x94decoding sequence according to the this method is as follows.
First, the sending side sends a signal designating a picture to be used as the reference picture at the same time as the encoded data, and the receiving side uses as the reference picture the picture designated by the signal designating the reference picture, and the received encoded data is decoded. In addition, the pictures which could be correctly decoded are sequentially stored in the reference picture memory at the receiving side. Next, the receiving side sends to the sending side the information concerning whether the encoded data could be decoded and information designating the newest picture in the reference picture memory. In addition, at the sending side, by encoding using a correctly decodable picture designated by the receiving side as the reference picture, the receiving side uses the picture stored in the reference picture memory as the reference picture, and it can be decoded without error.
Here, in the above-described xe2x80x9cinformation concerning whether the encoded data can be decodedxe2x80x9d sent to the sending side by the receiving side, there are two types of signal: the ACK signal and the NACK signal. First, the ACK signal is sent to the sending side by the receiving side when the picture is correctly decoded, and includes the picture designation data which designates a correctly decoded picture. In addition, the NACK signal is sent to the sending side by the receiving side when the picture cannot be correctly decoded, and includes picture designation data which designates the newest picture among pictures stored in the receiving side""s reference picture memory.
In addition, at the sending side, when the above-described ACK signal or NACK signal is received, the methods of changing the reference picture used in the encoding can be roughly divided into two types.
The first method switches from the reference picture used in the previous encoding to the picture designated by the picture designation data of the correctly decoded picture on the receiver side included in the ACK signal, or by the picture designation data of the most recent picture among the pictures stored in the reference picture memory on the receiver""s side included in the NACK signal. In this method, encoding is carried out by changing the reference picture so that the sending side refers to the picture which could be correctly decoded at the receiving side for every picture. This is called the ACK method.
In normal inter-frame prediction encoding, because the previous picture is always made the reference picture, when one picture has an error, the error will necessarily propagate to the next picture. However, in the ACK method, there is no propagation of subsequent pictures irrespective of whether or not the previous received picture is decoded without error because the reference picture is changed every picture. However, for each picture, the encoding efficiency is lower than normal inter-frame encoding.
The second method switches the reference picture used in the previous encoding to the designated picture by the picture designation data, included in the NACK signal, of the newest picture among the pictures stored in the reference picture receiving side memory. In addition, in this method, when a NACK signal is not received, encoding using the latest previous picture as the reference is carried out. This is called the NACK method.
In this NACK method, if there is an error in one picture, until the reference picture is switched at the sending side by sending the NACK signal, the error propagates like a normal inter-frame prediction encoding. When the picture with the error is not displayed, the propagation of this error will cause a lowering of the number of pictures that can be correctly decoded. However, the reference picture is changed only when a NACK signal is received, and thus the encoding efficiency is higher than the above-described ACK method.
The differences between the ways these two above-described methods switch the reference picture lies in the switching method of the reference picture at the sending side when receiving an ACK signal sent when the receiving side has correctly decoded the picture. Thus, there is a difference in whether to change the reference picture according to the picture designation data of the picture which has been correctly decoded at the receiving side included in the ACK signal, or to make the latest picture previously encoded at the sending side the reference picture.
In any case, depending on the aim of the communication, either the ACK method, which has the effect of decreasing the propagation of errors, or the NACK method, which has the effect of reducing the decrease in encoding efficiency, can be chosen, and this choice can be assumed before the commencement of communication and during communication.
Concerning the designating of the ACK method or the NACK method, when the designation is made at the beginning of the communication, it is possible for the receiving side to designate to the sending side in advance whether the ACK method or NACK method is being used. However, when the designation occurs during a communication, in the conventional methods, the sending side has been able to choose whether to use the ACK method or the NACK method, while the receiving side could not do so.
This is caused by the fact that what the receiving side designates to the sending side is only which specific picture is to be made the reference picture, and during communication, clearly, there was no method for the receiving side to designate to the sending side that the latest previously encoded picture is to be made the reference picture, and thus during communication the receiving side could not designate the NACK method, and there was no means to change from the ACK method to the NACK method.
Because of this, it was not possible to control the prioritizing of decreasing propagation of errors or reducing the decrease in encoding efficiency according to the requirements of the receiving side, by changing from the ACK method or to the NACK method during communication.
Furthermore, in the conventional ACK methods, when the memory capacity of the reference picture memory at the receiving side is small in comparison to the amount of the encoded and decoded picture data during the delay period, the oldest picture is deleted from the reference picture memory in order to store a newly decoded picture. Thus, there are cases in which the picture decoded within the delay time is deleted from the reference picture memory. When this happens, the reference picture is not stored in the reference picture receiving side memory, and it cannot decode the encoded data.
In this reference picture receiving side memory, the event which could not be decoded because there is no reference picture designated by the sending side, decreases the number of decodable pictures, and further decreases the efficiency of the inter-frame difference coding. Because of this, in the conventional methods, when the memory capacity of the reference picture memory is small in comparison to the encoded and decoded picture data during the delay period, changing from the NACK method to the ACK method was not possible.
Because the transfer of the encoded data and the picture designation data occurs via the transmission line, a delay occurs between the time the sending side encodes as the reference picture the picture indicated by the information for designating the newest picture in the reference picture receiving side memory sent from the receiving side and the time this encoded data arrives at the receiving side. Thus, at the receiving side, it is necessary to prepare for the case wherein after decoding the encoded data changed by the reference picture, this decoded picture is stored, and subsequently the sending side designates this picture as the reference picture.
In consideration of the above, one object of the present invention is to provide a method for instructing the sending side to encode using the latest previously encoded picture as a reference picture, as in inter-frame prediction encoding, and in this manner, provide an video communication method and a picture communication system along with a computer readable recording medium which records the picture communication program which makes possible the receiving side instructing the sending side to encode using either a specified picture as the reference picture or encoding using the latest previously encoded picture as the reference picture, and in this manner enabling principally the receiving side to carry out adjustment of the reducing of the propagation of errors or the reducing of the lowering of the encoding efficiency.
In addition, a further object of the present invention is to provide an video communication method and a picture communication system along with a computer readable recording medium which records the picture communication program which makes possible reducing the propagation of an error similarly to the ACK method, even in the case that the memory capacity of the reference picture memory at the receiving side is small compared to the amount of encoded and decoded picture data during the delay period of the picture data between transmission and reception.
According to the present invention, the above-described objects can be achieved by the sending side encoding an input picture by the inter-frame prediction encoding method, and sending this encoded data and the picture designation data designating the reference picture that was used during this encoding to the decoding side, and at the same time this encoded data is decoded, and this decoded picture is stored in the memory of the sending side as the reference picture. When a specified reference picture among the reference pictures stored in the memory of the sending side is designated by the receiving side, encoding of the input picture using this specified reference picture is carried out, and when the latest previously encoded picture is designated by the receiving side, encoding of the input picture is carried out using the latest previously encoded picture as the reference picture.
In addition, according to the present invention, when the decoded picture is stored in the receiving side memory as the reference picture and decoding of the received encoded data is carried out, a reference picture, which was designated by the picture designation data sent from the sending side and used for encoding the encoded data, is selected from among the reference pictures stored in the receiving side memory, and used as the reference picture to be used when encoding, the receiving side instructs the sending side, based on the specific condition as to whether to use a specified picture among the plurality of pictures decoded in the past at the receiving side as the reference picture or whether to use the latest previously encoded picture as the reference picture. At this time, when the sending side is instructed to use a specified picture among the plurality of pictures encoded in the past, the reference picture designation data specifying the specified picture is sent.
Here, instructions by the user of the receiving side, whether or not there is an error in the encoded data the receiving side received, or the degree of the data delay time between sending and receiving, etc., might be considered examples of the above-mentioned given conditions.
Whereas conventionally there has clearly been no means for the receiving side to designate a picture previously encoded as the reference picture and the sending side to use as a reference picture the latest previously encoded picture during communication, in the present invention it is possible for the receiving side to instruct the sending side to select as the reference picture to be used to encode the input picture the picture the sending side previously encoded by sending continuous reference picture designation data.
In addition, the above receiving side can also instruct the sending side to select the picture which could be correctly decoded by the receiving side as the reference picture used to encode the input picture by sending specific reference picture designation data to the sending side. That is, the receiving side can instruct the sending side to use the reference picture change method of the ACK method or the reference picture change method of the NACK method, and for example, depending on the communication conditions during transmission and reception, the receiving side can principally carry out the adjustment of reducing the propagation of errors or reducing the lowering of the encoding efficiency.
In addition, a further object of the present invention is achieved by the sending side having a sending side memory which stores a plurality of pictures encoded in the past as reference pictures, selects from this sending side memory a reference picture specified by encoded reference picture designation data sent from the receiving side, and using this selected reference picture, the input picture is inter-frame difference encoded, and along with this encoded data, the reference picture designation data specifying the reference picture used in this encoding is sent to the receiving side. Furthermore, the receiving side decodes the encoded data sent from the sending side using as a reference picture a picture, specified by the above reference picture designation data, selected from among a plurality of pictures decoded in the past and stored in the receiving side memory, determines whether or not this decoded picture is stored in the receiving side memory as a reference picture, stores in the receiving side memory only decoded pictures determined to be stored as reference pictures, and at the same time, makes the reference picture one picture from among the pictures stored in the receiving side memory used when encoding, and sends the data specifying the picture made this reference picture to the sending side as encoded reference picture designation data.
According to the structure of the above-described invention, because the reference picture designated from the sending side in the receiving side memory is always stored and can be referenced when decoding, even when the memory capacity of the receiving side memory is smaller than the amount of the encoded and decoded picture data during the delay period, until receiving the encoded data using as a reference picture the picture specified by the encoded reference picture designation data that the receiving side sends to the sending side, it is possible to continue storing in the receiving side memory, and because the reference picture used in encoding is always present in the receiving side memory, reliable decoding is possible.
Furthermore, in the present invention, the two above-described structures can be appropriately combined, and the effects obtained by each structure can be obtained simultaneously.
In addition, in the present invention, as a method for designating the latest previously encoded picture as the reference picture, the receiving side can send to the sending side a signal having a specific code. In addition, each time the receiving side receives encoded data, the receiving side can instruct the sending side which reference picture is to be used for encoding.
Here, the above-mentioned instructions for the sending side about the reference picture can be implemented by sending to the sending side either a continuous reference picture designation signal which instructs the sending side to use as the reference picture the latest previously encoded picture or the specific designated reference picture signal which instructs the sending dise to use as the reference picture a particular picture from among the plurality of past decoded pictures at the receiving side.